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Abstract: A user study is conducted to evaluate the efficiency and effectiveness of two types of visualizations to 
identify damages sites in case of disaster. The test consists out 36 trials (18 for each visualisation) and in each trial 
an area of lx lkm, located in Ghent, is displayed on a screen. This image shows the combined height information 
from before and after the disaster. The first visualisation, page flipping, is based on greyscale images with height in- 
formation from the pre- and post-disaster situation between which users can switch manually. The second visualisa- 
tion, difference image, is a result of subtracting the heights (before versus after) and assigning a blue- white-red col- 
our ramp. In order to simulate the urgency with which the data is captured, systematic and random imperfections are 
introduced in the post-disaster data. All participants’ mouse and key interactions are logged, which is further com- 
plemented by the registration of their eye movements. This give insights the visualizations’ efficiency, effectiveness 
and the overall search strategies of the participants. 
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1 . Introduction 


Disasters can struck anywhere and anytime, leaving a trace of destruction. Emergency response teams need to 
know the location of damaged sites as fast and accurately as possible in order to save lives. A promising option is to 
use the crowd, which enormous power has been demonstrated in response to the Haiti earthquake in 2010 (Zook et 
al., 2010). The fact that the volunteers not necessarily need to be present at the disaster site is a huge advantage of 
crowdsourcing: contributions can come from anywhere, anytime. Nevertheless, most of these volunteers are not 
necessarily experts in the matter at hand. Therefore, adapted tools and visualizations - easy to handle and interpret - 
are crucial. This research focusses on the evaluation of the efficiency and effectiveness of two different visualization 
in a situation where the crowd is asked to identify damaged sites in a crisis situation. 
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2. Study Design 


2.1 Participants 


Students, alumni, and staff from Gavle University College are invited to take part in this test. Their personal 
characteristics (age, gender, education, etc.) are obtained through a questionnaire, which also probes for familiarity 
with the displayed area. All participants participated voluntarily and received a lunch coupon. 


2.2 Stimuli 


The user study consists out of a training test followed by 36 trials (18 for each visualisation). In each trial an area 
of lx lkm, located in Ghent, is displayed on a screen. This image shows the combined height information from be- 
fore and after the disaster. It can be assumed that height information is available for most regions on the planet. Af- 
ter a disaster, height information can nowadays be gathered quickly using remote sensing techniques (LiDAR, pho- 
togrammetry, e.g. by the use of UAVs), as has been applied in response to the 9/11-attack in New York (Huyck et 
al., 2003) and the Haiti earthquake (Kent, 2010). In order to detect the damaged sites, selecting the correct visualisa- 
tion is crucial. In this study, two alternatives are evaluated. The first visualisation, page flipping, is based on grey- 
scale images with height information from the pre- and post-disaster situation between which users can switch man- 
ually (see Fig.l, left and middle image). The second visualisation, difference image, is a result of subtracting the 
heights (before versus after) and assigning a blue- white-red colour ramp (see Fig. 1, right image). Of particular in- 
terest is the effect of change blindness in these visualisations (Simons & Ambinder, 2005). 

In each of the images, damaged sites are simulated: buildings that have collapsed in different manners. The num- 
ber of damage sites varies between 6 and 8, but is balanced across all images. In order to simulate the urgency with 
which the data is captured, systematic and random imperfections are introduced in the post-disaster data. 



Fig. 1 . Visualizations of pre- versus post disaster height information using (left & middle) page flipping or (right) a difference image 


The stimuli are thus subdivided into two groups dependent on the visualization (page flipping versus difference 
image). Within each group, the stimuli are displayed in a random order to the participants. In order to avoid a learn- 
ing effect or influence due to fatigue, two measure were taken. First, each participant saw only half of the 72 stimuli 
(36 in Set 1 or 36 in Set 2). This way the participant never saw the same ‘area & damage sites’ -combination twice. 
Second, half of the participants started the experiment with the page flipping visualization whereas the other half 
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started with the difference images. Taking into account these measures, a rotation of the experiment conditions 
across four participants was implemented (see Table 1). 


Table 1. Rotation of parameters across participants to avoid learning effect and influences of fatigue 


PartNr 

PartID 

Start Visualization 

Data Set 

1 

1101 

Page Flipping 

Set 1 

2 

2102 

Difference Image 

Set 1 

3 

1203 

Page Flipping 

Set 2 

4 

2204 

Difference Image 

Set 2 

5 

1105 

Page Flipping 

Set 1 


23 Tasks and Recordings 


Participants have to indicate (by mouse click) the damage sites (6-8) as quickly as possible. When the page flip- 
ping visualisation is displayed, the participants can manually ‘flip’ between the pre- and post-disaster image using 
the left and right arrow keys. The participant can proceed to the next image by pressing the space bar. Nevertheless, 
a time limit of 75 s is set on each trial to simulate urgency and keep the study manageable. The system logs the time 
of each mouse click and keyboard interaction and the position of the mouse clicks. Furthermore, the participants’ 
eye movements are recorded to obtain insights in their visual behaviour (search strategies, interpretation difficulties, 
etc.). The experiment is implemented in a web site that is linked to the eye tracking software. For this experiment, 
the SMI RED250 was used to record the participant’s eye movements (see Fig .2). 



Fig. 2. SMI RED250, with difference image displayed as visualization 


23 Procedure 


The experiment follows a structured stepwise approach. First, the participant is given a general introduction on 
the context and goal of the experiment, the different types of visualizations and the tasks that should be completed. 
Second, (s)he is asked to fill out a questionnaire which focusses on personal characteristics (age, gender, level of ed- 
ucation, color blindness, familiarity with the region, etc.). Third, the participant could complete a short training task 
related to the first visualization. Fourth, the eye tracking equipment is calibrated before starting with the actual main 
test (which consists out of 18 trials). After completing this test, the participant could enter feedback on the test or 
visualization. Next, the procedure is repeated from step three and onwards for the second visualization. 
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3. Data Analysis 


Two key components complement each other in the analysis: the data gathered through the web site (mouse and 
key interactions) and the eye movements. The data from the web site is stored in a database from which it is export- 
ed. The available data sets are illustrated in Fig. 3. Using these data sets, comparisons between the visualizations can 
be made based on: trial completion time, number of correct demolitions indications, number of wrong demolition 
indications, time to first demolition indication, time between last demolition indication and trial end, etc. Further- 
more, part of the analysis focusses on the influence of the systematic imperfections on these measurements, com- 
plemented by, among others, the number of arrow key presses. 


Data Visualizations Results after tests 



Fig. 4. Data obtained from the web component 


An overview of the analysis scheme for the eye tracking data is presented in Fig. 4. The data is gathered by soft- 
ware provided by the eye tracking vendor SMI. However, this is a continuous recording of raw data over all trials, 
which has to be split up to have the data for each trial separately (by synchronising it with the res_trials data from 
the web logging). Next, the raw data is aggregated into fixations using the open source algorithm EyeMMV (see 
Krassanakis et al., 2014 for more details). These fixations are, on the one hand, used in statistical tests to compare 
between the different visualizations (based on among others Holmqvist et al., 2011; Duchowski, 2007). To obtain 
insights in the participants’ search strategies, on the other hand, we create (1) gridded visualizations (based on In- 
coul, et al., 2015) and (2) perform sequence analysis (based on £oltekin, et al., 2010). 
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Fig. 4. Data and analysis scheme for the eye tracking component 
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3. Conclusion and Future Work 


This study is a first step in order to evaluate which visualization is most efficient and effective to visually detect 
damage sites by the crowd in case of disasters. Furthermore, more insights in the participants’ search behavior is ob- 
tained - where do they start looking, is there a trend in how the image is scanned, which regions are neglected, etc. - 
is provides crucial information to propose improvements for these visualizations. Therefore, future work is foreseen 
to implement other visualizations (e.g. variations in the size of the area that is displayed, additional processing of the 
data, etc.) and characteristics of the participants (e.g. familiarity with the region). 
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